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How much Quality Can You Pay For? 

Bonnie C. Carroll 

How much quality are you willing to pay for? First let’s define quality. Quality is 
relative excellence, value for money, conformance to drawings and specifications, 
conformance to the purchase order, and fitness for intended function or use. There’s 
nothing absolute about all of these definitions. You have to create your own criteria for 
measuring quality. 

There has been an evolution in thinking about quality control. People used to look at 
quality control as statistical stamping, as asking at the end of the process, How good is 
the product? - then going back and fixing it. That’s quality assurance. But you need to 
know your operating procedures, you need to see the whole system, and you need to 
engineer the criteria and work processes in the beginning so that you will have fewer 
remakes at the end. You can combine quality assurance with quality control. This 
thinking evolved into total quality management (TQM) and that then to a continuous 
improvement process. It’s a different way of thinking about the whole notion of quality; 
it’s less statistical methods and product orientation and looking more at a people 
orientation and an empowerment of individuals to affect the work that they’re doing. 
This is the theme of TQM today. 


Quality in Databases 

What are the quality aspects of databases? At the very elementary level you have 
typographical quality - how accurate is it? Then you have the content quality of the 
record, scope and coverage (fitness for intended use), timeliness, and accessibility. 

Quality in the STI Program 

What are the quality issues in the STI Program? The issues are: internationally, we are 
getting beat; from a NASA perspective, the STI Program is not being used to the extent 
that we would like; individually, we’re not happy as employees, we’re not pulling 
together as a team, we’re not being fulfilled. 

The Deming Philosophy 

In the 1940s, W. Edwards Deming was a teacher of statistical techniques. He came from 
a very humble beginning; he had put himself through college. He believed that statistics 
were the solution to many industry problems. He taught statistical techniques to 
American government and industry in the ’40s. After the war, the United States wasn’t 
interested in Deming and his ideas. So he went to Japan at their invitation. There, he 
got 21 Japanese businessmen to sit around the table and buy in to his concepts of total 
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quality management. He initiated a new order in Japan - how to manage quality. The 
Japanese spent from the early ’50s to the ’80s preparing themselves to start beating us in 
the marketplace. In 1980, Deming was on the NBC program "60 Minutes," saying "If 
Japan can do this, why can’t we?" This presentation inspired the President of Ford, 
Donald Peterson, to call Deming and invite him into his office. Deming changed Ford’s 
corporate culture to "quality is number one," and thereafter a lot of other industries 
committed themselves to this whole philosophy of total quality management. 

What are Deming’s main points? He has fourteen points; this is a summary. 

Quality can cost less because you design it in rather than inspect it in. Current thinking 
about quality is that you have to constantly improve. 

You have to create a constancy of purpose. It means staying ahead of the customer, not 
only in meeting present needs but planning for future needs as well. That’s been the 
success of many U.S. industries; they have integrated total quality management into their 
business. Quality creates a new economic age for the today’s single global market. 

Mass inspection alone does not produce quality. Buying from the lowest bidder can be 
costly. Instead, to get supplies, according to Deming’s philosophy, you work with a single 
supplier to improve quality and lower the supplier’s costs, and in that development of a 
long-term relationship, all parties profit. This is a concept he promotes specifically, and 
of course the government acquisitions and contracting process tends to mitigate in a very 
different direction. 

When you work with the supplier, you get high quality the first time, which prevents the 
costly errors found later in the cycle. Deming makes the point that you want to develop 
a new basis of incentives. The way government contracts are awarded, by cost control, 
we’re monitoring the wrong things for what we’re trying to accomplish. You’ve got to 
change your philosophy and allow new kinds of contracting relationships so that you are 
paying people not just to meet budget and production quota but to do the right thing. If 
you can come up with minimum optimal specifications, you could in fact reduce costs. 

There are three rules of total quality management. The first isn’t Deming’s philosophy. 
You have to focus the organization’s energy on one single thing: total customer 
satisfaction. Don’t look at the product or the service, you’re looking at whether the 
customer is satisfied. Everyone on the production line needs to have this focus. Everyone 
needs to think, "What does this thing I’m doing have to do with achieving customer 
satisfaction?" - your ultimate measure. And if the answer is "Nothing," then find a way 
not to do it. 

Organize all work as a process. Focus on improving the process, not on the end result. If 
you do it right every step along the way, the end result is going to be right. That is the 
current Deming TQM philosophy. 
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The third point is give people a chance to seek continuous improvement for themselves 
and for the organization. 

In the service industry, quality is more difficult to measure. Therefore, it’s more difficult 
to comprehend this process. Customers don’t evaluate only on the service outcome. They 
also look at the process of delivering. If they get an article that is great, they focus on 
that and are pleased, but if they went through a hassle to try to get it, then the service 
quality is rated lower and their total impression is of lack of quality. The only criteria 
that count are the ones that are defined by the customers. If we’re abstracting or 
indexing and we re engineering in quality and the users never see it, they won’t care 
about that quality if it doesn’t contribute to what they see and their perception that it 
meets their needs. We have to have good authority files and so on that the users may 
not see, but we do have to think in terms of what the users are using that is important to 
them. Therefore, where do we put our emphasis in engineering quality? For instance, 
in looking at user friendliness and access, accessibility has been mentioned as an 
important value-added quality aspect of a database. 

Service quality is defined as the discrepancy between customers’ expectations or desires 
and their perception. Service expectations are formed by word-of-mouth communication, 
personal needs, past expectations, and external communications like advertising and 
literature. These factors define what the customer is looking for, what the perception is. 
The criteria that the customers use are tangible things: ease of communication, 
reliability, responsiveness - willingness to help customers and provide prompt service, 
competence - possession of the required skills, courtesy, credibility, security - freedom 
from danger or risk, access, communication - keeping the customer informed, 
understanding the customer - making the effort to know customers and their needs. 

There is a model that shows your expected service needs or personal needs. Your 
perceived service comes from expectations, and service quality is the net difference 
between the two. There is a standard methodology. Another model factors out from 
those original 10 dimensions of quality five things: tangibles, reliability, responsiveness, 
assurance, and empathy. These five items relate to the criteria we developed for 
database quality: access, responsiveness, and so on. 

In the Deming method, the bottom line is that TQM changes your relationship with 
customers, with suppliers, and with employees. This is a continuing theme. All of these, 
your external markets and your internal staff are also your customers; everybody on the 
assembly line is the customer for the next person; and your suppliers are also your 
customers in a sense that you need to negotiate with them. 

The pitfalls are that managers don’t understand economics, capacities, bottlenecks, and 
sources of delay. Do your management reports give you the data that you need to know 
what your economics, capacities, bottlenecks, and your sources of delay are? Some of 
the literature from the defense industry looks very specifically at the problems inherent 
in that industry where there is a lot of government contracting. Contracting relationships 
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focus on the wrong measures: cost recovery, not working together as a team to end up 
with satisfied customers. The fault is partly in the system - but if you can get supplier 
and contractor behavior to be quality-oriented and begin to develop a common 
understanding, then successful TQM programs find ways to recast all this reporting to 
truly reflect operating requirements. And you have to sit down and roll up your sleeves 
and develop a common understanding. You have to avoid over- or under-specification 
because it is costly and the only way to do that is when you understand what is targeted. 

Other pitfalls are that philosophies tell you to put quality in at the beginning, engineer it 
in, design it in, but your process is already in place. So how do you go back and retrofit? 
Retrofitting is very expensive. Also, you cannot automate your way to quality. You can 
become much more efficient but adding quality to the wrong processes is not the 
solution. Automation itself is not the solution. The "cruise control syndrome" addresses 
the lack of cross-functional integration. Something that is everybody’s interest, is 
nobody’s interest, and quality problems result from decisions never made. That’s STI’s 
problem. A problem that is pervasive throughout the whole research and development 
system is nobody’s problem. And yet if you look at the whole research and development 
system in a TQM context, you see that most of the processes that scientists and 
engineers go through are information processes. If information is part of every critical 
process, it really needs to be looked at more seriously. Embracing statistical process 
control is only one dimension in only one dimension. 

Another pitfall is paperwork and bureaucratizing the quality process. Successful total 
quality companies use less paper in their programs. TQM needs to be internal, 
individualized, and universal. No single action can replicate the accomplishments of the 
total quality company. Total quality is many small changes in culture. This speaks to 
competition, that comes later, but competitors can’t just jump in and catch up to a TQM 
company. TQM is a philosophy, a way of doing business, not just quick changes. And 
finally, visions don’t translate into action until workers understand the concrete changes 
required of them. People have to know how to use existing skills differently and how to 
gain new skills. The process really needs to be top down until the last person on the 
chain really understands. 

A United Auto Workers spokesman gets to the heart of the personal point of view: "No 
one likes to go home and say they made junk all day." That’s at the heart of TQM. 
People become proud and interested; you’re on your way to a quality company. 

There is a pyramid called the thought paradigm. From the top down are the top dog, 
vice presidents, middle managers, supervisors, and the workforce. Draw a line right 
above the workforce, and the people above the line are regarded as people who can 
think. People below the line are regarded as those who don’t or can’t think. The whole 
concept of quality control says, erase this line. That’s basic to this whole quality concept. 
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Conflict 

GSA has a workbook that looks at one way to implement TQM. It says that when you 
have quality teams and you try to empower people and have everyone involved, there 
are many opportunities for conflict. There are different types of conflict. One is 
differences in data or in information. A conflict based on having different data is an easy 
conflict to resolve because you can look at the data and deal with it. Strategy conflicts 
get a little bit more complex because strategy is less concrete. Strategy has to come from 
management, there has to be understanding and acceptance, so you get into more 
difficulty. If there are differences in understanding the basic goals, that is more serious. 
Conflicts in norms are cultural, because norms are the way you do business. When you 
have differences and conflicts in culture, you can really be booby-trapped. If you have an 
underlying basic difference in values, then that can really be landmined. If you think in 
these terms, you really need to be synchronized from the top management down and 
pass on the values, the culture, the goals, the strategies, and the data that you’re using to 
understand your system. 

Fad or Fact? 

Is this just another management fad like management by objectives, and strategic 
planning? Is total quality management a fad, or fact? Here are some examples that 
indicate that TQM is real, and is here to stay. 

At Ford, "Quality Is Job One." Donald Peterson, a Deming disciple, said, "Those 
enterprises that don t adopt a quality culture simply are not going to succeed." A lot of 
senior executives in the United States today believe this statement and are talking about 
it. Big companies are increasingly demanding that suppliers organize and follow the 
principles of TQM. One major money center is actually looking at a commitment to total 
quality as a determining factor in evaluating a company’s risk. It is coming into the 
whole business infrastructure in the United States. 

The Malcolm Baldrige Award for National Quality has become important to a lot of 
companies. This is a significant award and companies are competing for it. The award is 
oriented toward the Deming philosophy. The process refocuses the company and makes 
it look hard at itself. It gives companies a level of understanding of good and bad. To 
apply for this award, you have to rethink the company philosophy. The criteria for the 
Malcolm Baldrige Award are based on the Deming total quality principles. First, you 
need to have a plan to keep improving all operations continuously, then you need a 
system for measuring these improvements accurately, a strategic plan based on 
benchmarks that compare the company’s performances with the world’s best, a close 
partnership with suppliers and customers that feeds improvements back into the 
operation, a deep understanding of the customers so their wants can be translated into 
products, a long-lasting relationship with customers, a focus on preventing mistakes 
rather than just correcting them, and a commitment to improving quality that runs from 
the top of the organization to the bottom. These are the criteria that you have to write 
to in order to apply for the Baldrige Award. 
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In November 1989, the First National Total Quality Management Symposium was held, 
co-sponsored by the AIAA. 850 people attended, double the expectation. The highest 
levels of government and industry - the CEOs, the Assistant Secretaries - were 
represented and all were expressing their strong convictions that TQM is here to stay. 
Skeptics must discard doubts and embrace the concept or be left behind. 

Aerospace customers and suppliers, the STI Program’s users, or at least a large portion 
of the user community, are reading about TQM in every Aviation Week and Space 
Technology magazine. 

But what can you say against TQM? Changing to the TQM culture is a frustrating 
experience and expensive; it’s not a free ride. It’s a complete shift away from short-term 
perspectives. The CEO and bottom-line-oriented financial officers have to switch to a 
long-term view, which requires extremely strong outside motivation. Our culture is very 
short-term, bottom-line oriented. This thinking mitigates against this being a process of a 
system that really will have long-term impact as opposed to just a passing management 
emphasis. Change is inherently disruptive and will test the skills and intuition of even the 
most committed. Senior managers have to accept the timeframe and commitment 
required. Suppose, for example, that Gladys has been reading about TQM and she says 
you’ve got to have a long-term perspective - you’ve got to commit to my budget for five 
years so I can get this job done. This is what TQM says and this is what industry says 
you have to do; but how do you get from here to there? And smart competitors aren’t 
sitting still. 

Conclusion 

In closing, here’s an interesting statistic. Quality or price? What cost are people willing 
to pay? This was from Time magazine, 1989. A leading quality consulting firm in 
Massachusetts did a survey and concluded that while price still plays an important role in 
buying decisions, more customers demand high quality at any price level. Quality is more 
important than price. In 1978, 30 percent of the people said quality is more important 
than price. In 1988, 10 years later, 80 percent of the people said quality is more 
important than price. People are beginning to recognize the inherent cost of lower 
quality. 

Discussion Summary 

Some potential problems with TQM are that it can take up to 18 months for a company 
to do a feasibility study for adopting TQM; employees spend an inordinate amount of 
time in Quality Circle meetings; TQM is such a new way of thinking for most U.S. 
companies that adopting it is like adopting a foreign culture; with changes of ownership 
and frequent turnover of upper management, adopting TQM is difficult. 

Two companies that have adopted TQM successfully are Florida Power and Light and 
Xerox. Florida Power and Light employees seem to be happy, judging by their low 
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turnover rate. Xerox is so proud of its conversion to TQM that it prefaces all its client 
presentations with 20 minutes on TQM at Xerox. Xerox has also won the Baldrige 
award. Barbara Lawrence thought that applying for the Baldrige award was a mixed 
blessing. AIAA received excellent feedback in a debriefing session, but to continue to 
apply requires compiling statistical data on many levels for several years, which requires 
a lot of staff time. Apparently Admiral Truly wants to apply for the award for NASA in 
1991 or 1992. 

Adopting TQM does improve QA, because QA is part of the TQM process. Deming was 
a statistician; therefore measurement is an important component of the TQM process. 
Each team member participates fully, even in a service industry like the STI Program. 
Think of service as a product. 

Government-contractor relationships can incorporate TQM. Some are beginning to 
adopt the principles already, communicating wisely. 

What CASI Has Done To Improve Quality 
Joseph Gignac and Dian Marincola 

What has CASI done to improve quality? To select one example among many products 
and services, let’s concentrate here on database quality. Database quality is measured in 
terms of accuracy, completeness, timeliness, retrievability, and perception. These terms 
are not new. Whenever you think about database quality the first four come immediately 


Accuracy in the data is a very complex concept to deal with. It includes format, spelling, 

indexing - are we consistently applying the appropriate index terms, do the appropriate 
indexing terms exist? 

Next is completeness. If we’re supposed to have the premier database on aerospace 
engineering, do we have all the NACA reports? Do we have all the related literature? 
Are all of our records in scope? Were they in scope one time and are they now out of 
scope? Do you have everything you need to have? Can you plan to get even more - 
which also ties in very nicely with timeliness. 

Do you have the information in your database when you need to have it? And if you 

have long queues, is the information going to get out fast enough to be relevant to the 
R&D process? 

As for retrievability, you put records in the database so you can get them out again An 
error in a record will not necessarily affect retrievability. If you have a record with an 
item spelled wrong and that happens to be the word "the" you don’t really care because 
you are not going to retrieve on it. That misspelling matters for something else but it 
won t necessarily impact retrievability. Even more important than the obvious things like 
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correct spelling so that it can be retrieved properly, are you using proper formatting? 

Did you use proper conventions in indexing, etc.? What’s even more important is do 
you have enough fields so that you can retrieve the record properly? You re looking at 
the scope of the whole database for quality issues. 

It all comes back to perception. It doesn’t matter if all of these things are great, it s a 
great and accurate database, that we have run all kinds of spell-check and formatting 
routines before we put it into the system, it doesn’t matter if it s complete, that we hand- 
checked everyone of them, and we know that they’re all in there and timely. We have 
indexed everything, we’ve got full text - none of this matters if the user s perception of it 
is that it is garbage. And that is the hardest thing to remember when we re dealing with 
quality - the perception, from the user perspective. 

Database Quality Improvement at CASI 

Over the years CASI has done a lot to improve the quality of the database. Perhaps the 
most significant development effort that the Input Processing Section at CASI has 
undertaken is the machine-aided indexing (MAI) capability. MAI was established over 
about a 10-year period. It was probably in the neighborhood of a ten to fifteen person- 
year level of effort and at the moment it’s a real time-interactive tool. It doesn’t replace 
the indexing capability of CASI; it’s an assistant to the indexers to allow them to 
enhance the thesaurus term selections. In other words, the lexical dictionary, which was 
built over a number of years, is used as the base by which input from other agencies, 
such as DOE and DTIC, and actually the keyed abstracts, during the STAR input cycle, 
are done on an almost real-time basis. We’ve taken that capability and applied the 
NASA machine-aided indexing to all of the file series, such as the STAR series, the 
Limited STAR series, the unannounced, RTOPs, and so on. This has increased the term 
selection for retrieval purposes which Dian mentioned from an average of 8 to 10 terms 
per document to 10 to 12. This ties in with Deming’s philosophy of constant 
improvement. And it’s an improvement that could even be applied to AIAA. That’s just 
one of the most significant efforts. This system not only increases the term selection and 
focuses those selections on what is really relevant to NASA’s interest, but it is also used 
as a quality aid for input: our quality control staff checks titles and abstracts against 
terms in the lexical dictionary. 

I don’t know of any test of MAI against AIAA records. Again, this is an aid rather than 
a replacement. The next capability that we’ve undertaken more recently is the machine- 
aided duplicate check process. Heretofore, this has been a manual operation. After the 
cataloging is input, this is a machine validation check against the database and authors 
other reports that might be in the system, to minimize the duplicate processing activity 
which is very costly. And it eliminates the errors associated with the repetitive manual 
process. 

We have staff at CASI who are part of the CENDI cataloging standards committee who 
are working with the other Federal agencies to assure that the current cataloging 
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standards for standardized contract number format are viable and uniform across the 
agencies. More recently, machine-aided corporate source code selection capability was 
developed. This is an automated routine that looks up DTIC and DOE corporate 
sources and translates exact text matches to NASA corporate sources and codes in 
search code and tracks it to the IPS record. This is, again, not a replacement for a 
cataloging task but an aid. 

Another significant activity at CASI is continual updates to the database records. We are 
always receiving new or corrected information - the spelling of an author’s name, a 
corrected report number — and we change the STIMS records. 

In the post-STAR review, several key folks sit around and look at STAR and check to 
see whether the indexes are good and the abstracts are descriptive of the source 
document. These are all ongoing activities that strengthen and improve and enhance the 
quality of our products. 

Retrospective indexing is another interesting capability we have at CASI. As good terms 
are identified, old records in the database need to be changed. We also use the 
thesaurus. These new terms apply to the old file series. It’s not a static operation; we are 
constantly looking at new and better ways of improving the quality of the input. 

This applies not only to CASI - we have 25 years of experience with ESA European 
technical reports using the NASA standard formats. We’ve used those standards across 
the other government agencies such as CISTI, NASDA, ISA, and ADIS recently. We 
provide them the training, provide them the evaluation of their input, and provide them 
the feedback. Again, this is all part of the effort to ensure that what comes into the 
database is going to be standard and of high quality. 

Along those lines, we have developed a very comprehensive input processing manual 
that is a hands on "a to z" - how to prepare a record, what to do with it, where to send 
it, how to put it on microfiche, what goes on the masthead. This is for NATLEV. We 
did a lot of work in maintaining and updating the corporate source authority and the 
acronym dictionary. 

These are just a few of the things that are going on at CASI under the direction of NTT 
We’re constantly looking at better ways of improving the quality of input. I’m not saying 

it s all comprehensive, certainly there are areas for improvement and that’s what we’ll be 
continuing to look at. 


Quality from the User Standpoint (Abstract) 

George Roncaglia 

Paper products show improvements in turnaround times. Billing methods and procedures 
may also be improving, but data are insufficient at this writing. Reproduction quality still 
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needs improvement, using past year’s receivals as representative output. 

RECON is not keeping pace with the large-scale increases in SciTech publications. The 
amount of information going into RECON has declined while publishing levels have 
greatly increased. Net disparity between the two is wider than ever. Greater ease of use 
is important, but increased user-friendliness will be largely irrelevant without a more 
inclusive database. 

LaRC users would like to see RECON improvement, document conversion to electronic 
formats, and more creativity used in charge-back mechanisms. 


Database Quality: User Views Test Producer Perceptions 
Barbara Lawrence 


TQM is about continuous improvement, and that s the issue we face in addressing the 
NASA STI database. In considering the NASA database - lower-case database, meaning 
the generic database - regardless of whether it’s used in print in STAR or LAA, on 
RECON, or ESA, or DIALOG - on whatever output format, we have to think about the 
core of the data and the database. Quality has always been a hallmark of the NASA STI 
database, but user expectations change. That’s why we have to continually reassess what 
we mean by quality in this database. Let’s review our understanding of current user 
expectations for a quality database, and test the STI database against these perceptions 
and expectations. 

Database Records 

Primarily, I’m looking at the context of the records in the database. We all tend to view 
this NASA STI database from the output mechanisms that we are involved in. We have 
to step back from thinking about use and go back to the core, or the center, of the 
database. We’re talking about this today because quality has developed a lot of visibility. 
It’s a hot issue in the user community. We don’t think of quality so much in the context 
of a government database but a lot of these issues were raised by people saying, what if 
the databases are inaccurate? — we’ve got the author’s name wrong, we’ve put the 
wrong data in there somewhere. This is a litigious society; is somebody going to sue us? 
Or perhaps the database is not complete, not what we thought it was. You said it had all 
of these things in it and I didn’t find them. What’s going to happen? So the database- 
producing community realized it had better address these issues. 

Users use more than one database - not just all the files that are accessible through 
NASA/RECON, but all kinds of information - and they see a lot of examples, good and 
bad, a great variety, and they get new ideas and new expectations from all of that. 
Particularly in the NASA system, ours is a cooperatively built database in that the data 
come from many parties, and that number is likely to increase in the future. Now is a 
good time to think about what the current guidelines and standards for building this 
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database are so that when data come from Canada, from New York, from Baltimore, 
from Australia, they are all consistent with each other. 

I don’t know what the rules are for suing a government agency and maybe it wouldn’t 
apply directly here, but someone could sue a database producer or a database host if the 
data were inaccurate. This could happen if you put in an abstract "1" instead of H .l" and 
somebody built research based on that information. 

Obviously, quality is an attitude; not necessarily a matter of money but a matter of 
smarts. We need to work cooperatively, to talk to each other. I’m a strong believer in 
what I call working out loud." We need not only to have standards but to continue to 
review them to see that they meet current expectations. By standards I mean the internal 
processes as well as the formal external standards from ANSI and so on. We need to 
think about complexity versus simplicity. Sometimes we tend to make some things more 
complex than they need to be and maybe if we stepped back a little and ask ourselves 
whether we can do things more simply, we can improve our quality processes. 

User Interface 

We also need to recognize that the user sees database quality only in the context of the 
whole system. When I was at Exxon, we did everything we could to capture every 
document, every piece of technical correspondence, in or out of EXXON worldwide, but 
our user mechanism in those days was computer output, microfiche indexes, and full-text 
microfiche of the documents. Often at a given smaller location the manager wouldn’t 
spend the money to buy a microfiche reader and printer, he’d only buy a $150 reader. So 
here you have probably spent half a million dollars creating a database that had 
everything and people hated it because it was unfriendly to use. And then to read 
everything and not have a printer to compare the text from the document can be very 
difficult - so all that effort was almost for naught because as far as the user is concerned 
the whole system was just that microfiche. Ultimately, the searchable part of the 
database went online. We never did centrally divide the microfiche reader and printers 
for remote sites but we did a lot more work higher up in the management. We’d go to 
the management of that site, or of that division, and try to talk about why they needed 
to have more efficient user equipment. We may have the same problem with CD-ROM. 

What they’re doing now is capturing much more of the full-text data. The fiche readers 
from the early ’70s processed into full-text databases as well, but even in those days with 
these microfiche readers you loaded the fiche indexes into a cartridge so you could push- 
button your way through the searching of those indexes. It seems funny when you think 
of it that way, it seems kind of primitive, but it was a very large database for its time. 

I finally got to where I decided I’d better have one of these on my desk. If the way we’re 
disseminating information throughout Exxon around the world is on microfiche, I’d 
better get comfortable reading it, because if I can’t do it, then how can I possibly ask 
someone else to do it. 
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User Feedback 

The information for this presentation conies from our own contact with customers, partly 
through our customer service hotline, our 800 number, and particularly through the 
exhibits that involve both the information industry and AIAA technical meetings. We 
talked to scientists and engineers as well as information specialists. We also talked to 
people who are members of the AIAA Technical Information Technical Committee who 
are leaders in the corporate aerospace information environment. And generically, we ve 
participated in the National Federation of Abstracting and Information Services 
(NFAIS). Groups that are working in the area of quality have a working group which I’ll 
talk about a little later to define database quality issues. We’ve been involved with the 
Southern California Online User’s Group (SCOUG), and a workshop they held last 
summer where they defined database quality characteristics. I also looked at the 
literature and I have a bibliography, not comprehensive, but just some things that I have 
found interesting on database quality. 

Quality Needs 

Next let’s look at quality needs of users and where these needs come from. The 
intermediaries are the most specific about what they want. The interesting part of that is 
not only do they want it all but they seem to keep redefining the "all." That’s the 
challenge for us in this room: how do we keep up with that "all" and how do we judge 
what’s reasonable and what’s not reasonable, and what’s cost effective and within our 
means and abilities? Another message that users have sent is that they want the 
database producers to cooperate with each other as well as wanting the database 
producer and host systems to cooperate. NASA is unique in having the ability to call the 
producers together: the AIAA, RMS, whoever it is, you can pull them together and talk 
about what database quality is and get them to work together. And at least in one of the 
using venues you’re also a manager of the host system so you have a way of making sure 
that those processes talk to each other. End users are much less concerned with the 
specific sort of things intermediaries mention. They don’t want jargon but they do want 
consistency. They want to know what they’re getting. They need systems that will help 
them with search strategy and database-seeking guidance but they don’t want to know 
how they get there, they don’t want to learn about database file structures, they don’t 
want to know about statements of coverage and scope, they just want to know, "Is this 
the database that I need to get where I want to go?" The specific group of users that we 
all worry about, aerospace scientists and engineers, understand and are pretty 
sophisticated as a user community. They understand the concepts of databases and 
information retrieval. They’re not intimidated by technology, their needs are for very 
specific problem-solving information. That seems to be confirmed over and over again 
with conversations that we have with our findings, from Tom Pinelli’s data, and so on. 
And they need that information on their timetable. They don’t want to wait for an 
intermediary in the library to have time to do a search for them - they’s rather do 
without the information than wait a week for it. Aerospace scientists and engineers will 
do the searching themselves and they will learn a complex system although they would 
rather not. But if they’re not pleased with the service quality, with the database and the 
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system that database sits in, they’ll find another way to do it. For example, Paul Zarchan 
is a scientist, an AIAA member at Draper Labs, and they were doing an experiment of 
end user searching and they wanted him to do all the searching in the library and he said 
I won’t, no way. Then I’m meeting your timetable, your requirements, even if you’re 
paying for all the searching and all the databases and everything else. He said, "You give 
me one lesson and give me a password. I’ve got my portable PC and I’m going to take it 
wherever I go and see what it’s like to use information all the time." He went away and 
that’s what he did. It was on his terms for his needs. He needed not only scientific and 
technical information but newspaper information and business information. He was on a 
meeting program session we had in Reno with Roger Summit, he wanted to know who 
was Roger Summit anyway. So he went and looked him up on a Who’s Who database 
and found out who he was talking with on the program. 

Problem Solving 

Problem solving addresses issues of indexing, particularly, and I have a lot to say about 
indexing. How do you get to the right level of specificity in a database as comprehensive 
and as broad as the NASA STI database? The quality of the abstract is another thing. 
There I think we stand up very well. The abstracts are truly informative abstracts in this 
database from all sources and that turns out to be very helpful. Some of it can be in the 
software end but that’s one of the issues that we face. If we only deal with the problem 
at the retrieval software end, then that’s only going to help the people on that database 
system, and this database is used in a variety of systems. If you do an improvement at 
the RECON end, that’s terrific for all the RECON users, but it does not penetrate the 
data which are then used in a number of ways. Some of the folks who have been talking 
about database quality are Tom Aitchison, who initiated the whole rebuilding process for 
the INSPEC database. This has been one of the most impressive efforts in our 
community for a long time. It was a 3-year process. They rethought everything. Did they 
need all the fields, did they need new fields, did they have to retrospectively index old 
documents? They treated all the records from beginning to end. Jo Maxon-Dadd has 
tremendous experience both as a database user and from her Lawrence Livermore days 
as a database teacher at DOE. Now with 5 or 6 years at DIALOG, being responsible for 
the Sci-Tech databases, and having worked with databases of every size and shape and 
quality, she has a lot to say. Ann Mintz is in the business community but is one of the 
most thoughtful database users and articulate in terms of describing database quality 
criteria. In a workshop last summer we spent three days trying to define database quality 
characteristics. SCOUG has picked up on that work and is working on four codes of 
practice for database producers. One is standard database description, the second is a 
code describing interaction between database producers and vendors, the third is a code 
of practice for customer interaction with the producer or vendor, and the fourth is 
database quality per se. They’re trying to define measurable or definable characteristics 
of databases, and how to implement a rating system. 

NFAIS decided they didn’t want to get involved in a rating process. Publishing all these 
ratings has gotten highly complicated. Each database on each host system needs a 
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different rating. It seemed too complicated so we just created a set of guidelines. And 
there are so many variabilities. What’s timely for an STI database is completely different 
from what’s timely for a stock-market database. 

Retrieval Environment 

The information retrieval environment has changed a lot. The tools have changed, our 
experience base has changed. In 1972, all we expected was to have a database. That 
meant quality. Later on quality was defined as improving the searchability of those 
databases by adding abstracts, because not all the files originally had abstracts loaded. 
Then it was making all the fields searchable. Then, looking at some things like the 
Union List and the Battelle online system which was commercial for about six months. 
They loaded NTIS without the abstracts. Dialog had it with the abstracts. They loaded 
one database with a stopword list that was any word fewer than four letters so we in 
Exxon could not search for oil, gas, tar. They didn’t stay in business long. Then database 
users worried about errors, particularly basic typographical errors and wrong codes and 
so on and Compendex was the database we loved to hate. We used it because it was 
necessary but nonetheless everybody loved to complain about it. 

Indexing 

Then we thought more about index quality. The people here might be more familiar with 
the National Library of Medicine and the whole structure of the MeSH system. I was 
obviously a user of the American Petroleum Institute database and that had a very, very 
controlled structure. API’s is not really a controlled vocabulary. It’s better than a term 
list but it’s not fully controlled. One feature they had was called autoposting. How can 
you get to the right level of problem solving? People indexed, as we all do, at the most 
specific level of the item. But all the terms above that in the hierarchy were 
automatically coded into the database so the searcher could go in at any level. If you 
wanted animal you’d get cat and dog and animal and rat and rabbit. If you wanted 
rabbits, you’d only get rabbits. And if you wanted long-eared rabbits, then you could go 
and search for long-eared rabbits. The searcher had good retrievability and reliability in 
their system. Other approaches to indexing show how complex getting it right is; for 
example, the ABI Inform business database. When it started it was interesting to a lot of 
people because there weren’t any other business general-magazine kinds of databases out 
there. However, it didn’t have any indexing, it was just free text, the words and titles and 
abstracts. Business language has a very uncontrolled vocabulary, unlike scientific 
language. People demanded indexing so they created a machine-aided indexing process, 
and indexed the whole file. It was garbage. They had to throw all of that away a couple 
of years ago and then go back and manually index everything. It had to do with the 
structure of the kind of information, the kind of language in that literature, and 
manually was the only way they could do it. Now it’s one of the most highly regarded 
and highly used databases in this country, being loaded on the systems and at all kinds of 
universities because it does serve a purpose - but it wasn’t a quality database when it 
started. 
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Information Use 

To evaluate the effectiveness of information retrieval, consider the integration of the 
results into the user’s own life as well as the evaluation of the usefulness of the 
information for the resolution of the problem. This addresses not only relevance in recall 
in a search - not just whether we found information that the users wanted to solve the 
problem - but did they in fact use it. We’ve now raised the measure of quality one step 
higher. Can they use it, did they integrate it into their lives? 

Defining the Question 

Our systems are geared to answering well-defined questions. In the last few years we 
have improved the retrieval system so that the databases are easier to use, and we can 
help people through that process better. However we’re not as good at the earlier phase 
of the information-seeking process, the definition phase. Most people don’t start off with 
a very specific question. Sometimes they do, particularly in engineering. You want to 
find the standard or the specification for x, you want to know if there’s a widget for y, 
you want to know if somebody has designed an algorithm to calculate the flow in a 
certain place, whatever it is, you have a specific question - but often you don’t. You’re 
just faced with something and don’t know where to start and our information systems 
don’t really help people very well with that. 

Characteristics of Quality 

The generic characteristics of quality on my list are accuracy, comprehensiveness, 
consistency, timeliness, accessibility, support, cost/value, and TQM. The first four 
characteristics are the ones that deal with database quality per se and my definitions are 
no different from the previous ones. Accuracy means error rate, error correction, 
treatment of errata. Comprehensive applies to scope and coverage, gaps, source 
evaluations, source quality, duplication. Consistency. Do we use guides and standards? 
Are we consistent over time? Do we do what we say we’re going to do? If you say you 
have 100% abstracts, do you have 100% abstracts? The next three qualities relate to the 
system as a whole. Accessibility, support, and cost and value. TQM is on the list because 
we won’t have any of the others unless we manage intelligently; we need quality 
assurance and quality control processes. Are our staff trained to understand what we 
want out of the system? Are they empowered? Do we embrace the philosophy of 
continuous improvement? Do we evaluate what we’re doing on a regular basis? Do we 
see what others are doing and learn from them? 

First, let’s look at the EUSIDIC guideline on correcting and updating (or the draft), 
then we’ll take these characteristics and look at the NASA STI Database, and I’ll give a 
very personal assessment of how I think the database may stand up. 

In the area of accuracy the European Association of Information and Dissemination 
Centers (EUSIDIC) has issued a draft guideline on correcting and updating. A lot of 
these guidelines have to do with the interaction between the user and the database 
producer. Guideline One: Organization and Supplying Data, should indicate to the 
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customer the organization’s policy regarding data changes, modifications, or corrections. 
Guideline Two: Any organization making changes to existing data should always inform 
all organizations delivering the data to third parties. Guideline Three: The normal 
method of correcting data should be by adding an addendum to the existing data rather 
than replacing the record. In all the commercial databases and in the government 
systems, we replace an old record with a new record. It would be a radical change 
around the world if we followed Guideline Three. Guideline Four says that all 
collections of data should make explicitly clear the date of the last update and the most 
recent information that is in that file; if it says the file was updated on June 15th and the 
newest record is May 29th, people won’t go looking for stuff they can’t find. Guideline 
Five: Where practical and possible, all collections should make explicitly clear the latest 
time and date for which data can be found. This draft doesn’t address all the issues, just 
a few specific things. This draft is now circulating for comments and by the end of the 
year will be issued in some form or another. 

I think this NASA database stands up very well in terms of these quality characteristics, 
but the user expectations have changed over time. As more of the international parties 
get involved it’s a good time to think about what we’ve learned and see if we need to 
review, modify, and upgrade our guidelines and do some work on the older data in the 
database. In terms of accuracy, for instance, we all correct errors as they’re found in the 
form that shows up in RECON. We basically go back and correct the STIMS record so 
RECON users get revised data, whether it’s a typo in a title, an incorrect code, or 
whatever might be found. But if those corrections are made after the date of publication 
of the IAA STAR issue, the users of the print version only do not even see those 
corrections. 

Our record checking is almost all automated. The quality assurance processes and the 
validation processes are all automated, both at AIAA and at CASI. But there are errors. 
Particularly, in the olden days, errors occurred in the abstract. They occurred in 
transmission, or in transformations from system to system, particularly with some of the 
kinds of hardware that were used once upon a time. The LNS file goes back to the early 
1960s, and there’s material in there that’s messy. 

In the area of comprehensiveness, in terms of scope and coverage, the database does 
stand up very well. One thing that is very special about this file is that there is no gray 
literature: it has all the document types, it has the technical report literature, it has the 
meeting paper literature as well as the journals and the books and all the pieces. That’s 
a strength and a breadth that very few databases have. We do review sources for quality. 
Does this journal, for example, cover the same kinds of things that we now are looking 
for? We all have faced resource limitations and have some different acquisition 
approaches. That’s been a big issue with Europe. What they think they need and what 
we define as scope and coverage weren’t the same things so now we’re trying to rectify 
those issues. Some parts of the input community to this database are more aggressive in 
acquisitions than others. Some of the European users didn’t put in all the documents 
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that they might have because they only had to put in one for every hour of use. That 
kind of approach and philosophy sets a policy issue that will change the completeness of 
European acquisitions. However, the general sense is that the basic approach towards 
comprehensiveness in this file is very good. 

In talking about consistency, let’s look at the thesaurus as an example. It’s a very good 
thesaurus. It has good inclusion of terms. There’s a lot of information that’s in the 
printed tool, the access vocabulary, that has the data, the term, and the definition that 
isn’t in any of the online systems and that would be valuable to users. We have to think 
about whether we update the terminology often enough or not; particularly, the 
retrospective indexing which you said you’re now doing at CASI. But are the other 
people who use the same records getting that new data so that all the users of the data 
are seeing that improvement? 

Duplicate checking is an important consistency and accuracy process. The structure of 
the thesaurus allows for good searching. This thesaurus does a very good job of covering 
not just the scientific and technical terms but the mission terms, the systems terms, the 
properties and characteristics, and the different sorts of scientific aspects of this 
literature. The fact of having major and minor terms makes for good searchability. The 
level of indexing to give you a good balance in terms of recall and relevancy. There 
aren’t too many terms. If you have a database that indexes everything that you might 
think of, then you get a lot of garbage when you do a search. If you don’t have enough 
terms, you don’t find the things you need. But I suspect those guidelines, and the 
application of how we approach indexing, are quite different among the providers. We 
need to evaluate the machine indexing against the STAR records versus the IAA, not 
just in terms of machine indexing, but to find the differences. 

Considering the use of codes, I’m sure that we all do the right coding but our coding 
systems may no longer be the right systems. Names of countries change. We have had a 
big debate about whether you change the codes or not, and for the moment we’ve 
decided to look at this from the point of view of retrievability. What if you changed all 
the codes that said Germany, the ones that used to say East Germany and West 
Germany in the country of origin field? You can change the country code to Germany, 
but the author affiliations will still say East Germany or West Germany or wherever the 
author was at the time the article was written. You’d preserve the integrity of the 
information but you’d also preserve the retrievability. 

In the records in the database some fields exist only to create various print products that 
may or may not still exist These fields are no longer necessary. The file structure of this 
database is based on some combination of print-driven philosophy and the computer 
technology of the early ’60s. There is a lot of funny stuff in there. I looked at three other 
questionable data elements, not just the country code. We should reconsider some fields. 
In the author and author affiliation fields, our policies are to put the NASA authors first 
and to group the author affiliations, so that if the first author and the third author are 
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from the same organization, we move the third author to second. Well, this isn’t how the 
users want it. The scientists and engineers want the online document to look like the 
print document, and our method is very confusing to people. And, if you talk about 
doing a duplicate check between this file’s post-processing mechanisms and others out 
there that create the print version, we have certainly created a problem for users in that 
respect. There are some other differences in author fields in the way of punctuation, but 
the sequencing is one of the bigger issues that we’re going to look at. We’ve discussed 
subject categories. The country codes: do we update, do we not, what do we do with 
those old records, at what point do we all agree, all the people who are entering data in, 
to start using a new code? Do we wait until there’s a national coding system, do we 
read the newspaper and get on the phone and call Jim Erwin and say how do we make a 
decision about this? How are we going to deal with those things? The database 
structure was originally designed so that the smallest data element intellectually was also 
a data field. Now we tend to combine them, and when we try to create algorithms to 
parse them out for better retrievability, it’s very hard to do. We can’t always do it. It 
would be good to separate them again for publishing purposes - in our system we create 
the journal volume, issue, and all that stuff separately - but when we get the data back 
from STTMS it’s all together again. That’s another issue to consider. What’s the right 
way to do it for all of us? 

New Fields 

Some fields that users tell us they’d like us to think about are ISBN, CODEN, treatment 
codes, specialized indexing, pre-1972 abstracts, and trade names. We do have ISSNs for 
serials but we’d like to go back and add the old ones. We’d like to add the ISBNs for 
books. Treatment codes is a current topic that other sci-tech databases want. They tell 
the user whether an article is about theory, has experimental data, is about design, is a 
review article, or is a policy article. It’s another way to cut the database so that you get a 
better kind of answer, better precision. There’s specific indexing for specialized purposes, 
the kind of project that’s starting today, such as the National Space Science Data Center 
(NSSDC) indexing, where six journals are going to be selectively processed with a coding 
system that meets the needs of that community. Is the database designed to 
accommodate that kind of specialized process? Users also ask to have the older 
abstracts loaded. It seems as though it could be managed with the scanning process, but 
I know that database producers who have experimented with that have gone another way 
and are using offshore keying to get at the old data. Some users also want trade names. 
The NASA policy for the thesaurus is not to add commercial trade names, yet users 
would like to have access to them. Should they be included as index terms or in the 
thesaurus? That’s another policy question. 

Fields to Delete? 

There are some fields to reconsider. First, corporate source codes. Since the systems can 
now search by name and since the corporate source codes, as they’ve been used over the 
years, have changed, the same code gets reused for different organizations and they 
don’t pyramid. Imagine a system where code 123 is Boeing and code 1234 is Boeing 
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Military Aircraft and 12345 is Boeing Military Aircraft in a specific location, so you can 
search at various levels of specificity. Why do you need to do that when you can now 
full-text search on the corporate source field? That would save a whole lot of us a lot of 
headaches in trying to keep these codes up to date. When the data we create are fed to 
CASI, we see from their validation processes that most of the errors come from 
corporate source codes that have just been issued in the last few weeks. Then there are 
sales prices. By the time you publish them, vendors probably have changed them. Again, 
this is from a print publication, when people primarily read this abstract bulletin and 
looked at it for new books, and that’s not how the information is used any longer. A lot 
of codes are internal to NASA to produce specialized print bulletins. But we don’t need 
all these codes anymore. Another example is the number of analytic subsidiaries. If 
you’ve got a conference entry, does anybody use this information? 

Evaluating Data 

We should sit down as a team and evaluate whether we need all the data we have, and 
whether we are not collecting data that we should be collecting. I think we have a good 
database, but in keeping with the philosophy of continuous improvement, there are 
questions to be asked. Maybe we’ll make changes, maybe we won’t. But this is a good 
opportunity to raise the questions, particularly as we have multiple organizations 
responsible for creating data that go into the file - and their number is clearly going to 
increase as time goes on. The objective of the quality database is to provide the right 
information and information that is easy to use at the same time. If it’s the right 
information but hard to use, it doesn t make any difference because nobody is going to 
use it. We all know how to do it if we look at the definitions and guidelines. We have 
the tools to create clean data, to update our input guidelines, and to ensure consistency. 
We have quality assurance and quality control processes and we can share them with 
each other. I think we do a very good job of that. The issue for us is a matter of 
cooperation and intelligence. It’s not a matter of money, it’s not a matter of things that 
are impossible or difficult or driven by the technology. It’s a matter of sitting down and 
addressing the questions together. 
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Discussion Summary 

Initial action to take to assure database quality will be to form a working group 
consisting of representatives from NASA CASI/RMS Associates, AIAA, NASA 
Headquarters STI Program, and RECON users (information intermediary or scientist). 
This group will meet and draft a strawman document to be circulated to the rest of the 
STI Program community for comment. Barbara Lawrence feels that this initial step could 
be completed in two 2-day sessions. Jean Tolzman stated that it was important to define 
terms before doing anything else, and draft a statement of intent of the database: What 
is this database whose quality we want to improve, and what is it for? Who are its users 
and what do they use it for? To what extent is its structure defined by print technology, 
and should it be? 

Barbara Lawrence said that initially the working group should concentrate on the 
structure of the database, and particularly the structure of individual records. Dian 
Marincola suggested they concentrate on the "Aerospace Database STAR and LAA 
— because their record structures are similar to each other and to those of several 
related files. Geoff Worton felt that they should address scope and coverage first. 
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The document created by the working group should identify incremental tasks to be 
accomplished to attain a quality database, and should then rank these tasks in priority 
order. 

Some specific tasks mentioned were 

looking at fields in records and assessing which ones to retain, to delete, to revise 
or to condense; 

consulting with INSPEC about the recent overhaul of their database; 

thinking about initial actions that could be taken such as emulating Penn State’s 
"Oops" command that allows users to comment on individual citations online; 

keeping all organizations that supply data in mind when considering changes to 
format; 

looking at how records are selected for input into the database; 

looking at the relationship between the database and the print media it generates 
(, STAR , IAA ); 

looking at how the data are used — what do the users do with the information they 
find? 

considering whether to change Country of Origin entries to reflect historical 
change; e.g., East Germany and West Germany could be changed to Germany; 

considering mapping new subject codes to former ones; 

remembering that what is wanted is a process for assuring continuous 
improvement of the database, not just one-time changes. 
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HOW MUCH QUALITY CAN 
YOU PAY FOR? 


NASA STI Coordinating Committee 
"QUALITr 
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DEFINITIONS OF QUALITY 


1 . Relative excellence 

2. Value for money 

3. Conformance to drawings and specifications 

4. Conformance to the purchase order 

5. Fitness for intended function 
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EVOLUTION OF QUALITY THINKING 
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QUALITY ASPECTS OF DATABASES 


• Typographical quality 

• Content quality of record 

• Scope and coverage 

• Timeliness 

• Accessibility 


SLIDES 1:BANASA8 1.701 


WHAT’S THE ISSUE 


Internationally - We’re getting beat 
Institutionally - We’re not being used 
Individually - We’re not happy 
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DEMING QUALITY -STORY 


• 1940’s Taught statistical techniques to American 

government and industry 

• 1 950’s Initiated a new order in Japan - How to 

Manufacture Quality 

• 1980 NBC 90 Minutes "If Japan Can, Why Can’t We?" 

• 1 981 Don Peterson at Ford called Deming 

"The Rest is History" 
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DEMING’S MAIN POINTS -1 


• "Quality can cost less because you design it in rather than 
inspect it in." 

• "Improve constantly" 

• "Create constancy of purpose" 


Means staying ahead of the customer, not only meeting 
present needs, but planning for future needs, as well. 
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DEMING'S MAIN POINTS -2 


• Creates a new economic age with single global market 

• Mass inspection doesn’t produce quality 

• Buying form the lowest bidder is costly 

• To get supplies, work with single supplier to improve Q and 
lower his costs, developing long term relationships in which 
both profit 

• Statistical methods are a means to understanding 
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3 RULES OF TQM 


1. Focus organization’s energy on one single thing: TOTAL 
CUSTOMER SATISFACTION 

Everyone on production line 

What does this thing I’m doing have to do with 
achieving customer satisfaction? 

If nothing find a way not to do it. 

2. Organize all work as a process 

Focus on improving processes not on good results. 

3. Give people a chance 

Seek continuous improvement for themselves and the 
organization 


8UO£S1:BANASA8 1.701 


SERVICE QUAUTY HAS 
SPECIAL CONSIDERATIONS 


• More difficult for customers to evaluate than goods quality so 
more difficult to comprehend. 

• Customers don’t evaluate only on service outcome - also 
process of service delivery. 

• Only criteria that count are the ones defined by customers 
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SERVICE QUALITY IS DEFINED AS THE 
EXTENT OF DISCREPANCY BETWEEN 
CUSTOMER EXPECTATIONS OR DESIRES 
AND THEIR PREPERCEPTION 


SERVICE EXPECTATIONS ARE FORMED BY: 


• Word of mouth communications 

• Personal needs 

• Past expectations 

• External communications 


Zeithaml 



DIMENSIONS OR CRITERIA 
FOR DEFINING QUALITY 


Dimension and Definition 


Tangibles: 

Appearance of physical facilities, equipment, 
personnel, and communication materials 

Reliability: 

Ability to perform the promised service dependably 
and accurately 

Responsiveness: 

Willingness to help customers and provide prompt 
service 

Competence: 

Possession of the required skills and knowledge to 
perform the service 

Courtesy: 

Politeness, respect, consideration, and friendliness 
of contact personnel 

Credibility: 

Trustworthiness, believability, honesty of the 
service provider 

Security: 

Freedom from danger, risk, or doubt 

Access: 

Approachability and ease of contact 

Communications: 

Keeping customers informed in language they can 
understand and listening to them 

Understanding 
the Customer 

SLIDES 1 :B ANASAB 1 .701 

Making the effort to know customers and their 
needs 




CUSTOMER ASSESSMENT OF SERVICE QUALITY 
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CORRESPONDENCE BETWEEN SERVQUAL 
DIMENSIONS AND ORIGINAL TEN DIMENSIONS FOR 
EVALUATING SERVICE QUALITY 


Original Ten 
Dimensions for 
Evaluating Service 
Quality 

Tangibles 

Reliability 

Respons- 

iveness 

Assurance 

Empathy 

Tangibles 






Reliability 



Responsiveness 



Competence 

Courtesy 

Credibility 

Security 



Access 

Communication 

Understanding the 
Customer 
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Deming Method - TQM 


Changes the relationship with 
customers, 

with suppliers, with employees. 
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PITFALLS ON THE 
ROAD TO QUALITY 


• Managers don’t understand economics, capacities, bottlenecks, 
sources of delay 

Data often not available 

• Contracting relationships focus on wrong measures 

Contractors systems focus on cost recovery 

• Supplier/subcontractor behavior needs to be Q oriented 

Have common understanding 

• Avoid over or underspecification - it’s too costly 

• Successful TQM programs find ways to recast reporting to reflect 
operating requirements 
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A lot of what we’re doing to 
prevent dishonesty is counter 
to the basic notions of Total 
Quality Management. We’re 
trying to inspect honesty into 
the defense industry today. 

We have 26,000 auditors out 
there inspecting it in. 

Aerospace Business 

Aviation Week & Space Technology/12/4/89 


PITFALLS 

(Continued) 


Processes are institutionalized 

- Greatest opportunities to improve Q are initial front end design 
Can’t Automate your way to Q 

Adding Q to the wrong processes is a loser 
Cruise Control Syndrome - lack of cross function integration 
Q problems result from decisions never made 
Embracing SPC in only one dimension exacerbates the problem 
Paperwork and Bureauritizing Q process 

Successful TQ companies use less paper in programs 


PITFALLS 

(Continued) 


• No single action can replicate accomplishments of a TQ company 

Many small changes change culture 

• Visions don’t translate into action until worker understand concrete 
changes required of them. 

Use existing skills differently 
Gain new skills 
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"No one likes to go 
home and say they 
made junk all day." 

- UAW Spokesman 


Technology Review 
7/20/90 
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QUALITY TEAM DEFINITION 
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A high performing group in which the job 
gets done with excellence and the people 
feel included, valued and appreciated. 



TYPES OF CONFLICT 
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FAD OR FACT 


Ford "Quality Is Job 1 " 

Donald Peterson retired Chair is Deming disciple 

"Those enterprises that don’t adopt a quality culture simply 
are not going to succeed." 


Big companies are increasingly demanding suppliers organize and 
executive along, these principles. 

One major money center back is looking at commitment to 
TQ as determining factor in evaluating company’s risk. 
(Oliver) 


Malcolm Baldridge National Quality Award 

"The process refocuses companies and makes them look 
hard at themselves. Gives them a level of understanding of 
what’s good and bad." 



8 CRITERIA FOR MALCOLM 
BALDRIGE 

NATIONAL QUALITY AWARD 


A plan to keep improving all operations continuously. 

A system for measuring these improvements accurately. 

A strategic plan based on benchmarks that compare the company’ 
performance with the world’s best. 

A close partnership with suppliers and customers that feeds 
improvements back into the operation. 

A deep understanding of the customers so that their wants can be 
translated into products. 

A long-lasting relationship with customers, going beyond the 
delivery of the product to include sales, service, and ease of 
maintenance. 

A focus on preventing mistakes rather than merely correcting them. 

A commitment to improving quality that runs from the top of the 
organization to the bottom. 


- Curt W. Reiman 
Fortune 4/23/90 




FAD OR FACT 
(Continued) 


• In DoD aerospace the commitment shows no signs of waning 


• November 1989 "First National Total Quality Management 
Symposium" 

DoD 

AIAA 

National Security Industrial Association 
American Defense Preparedness Association 

850 attended - double expectation highest levels of government 
and industry. Aerospace and defense executives stressed strong 
convictions that it’s here to say 

Skeptics must discard doubts and embrace the concept or be 
left behind. 


• Aerospace customers and suppliers are hearing about it 
Aviation Week and Space Technology 
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FAD OR FACT 
(Continued) 


BUT, CHANGING TO TQM CULTURE IS FRUSTRATING AND EXPENSIVE 
WITH HIGH FRONT END COSTS 

• Complete shift away form short-term perspectives 

• CEO and bottom line oriented financial officers switch to Iona term 
view 

Requires strong outside motivation 

• Change is inherently disruptive 

Test skills and intuition of even most committed 


• Sr. Managers must accept time frame and commitment required 


And smart competitors don’t sit still 
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QUALITY OR PRICE? 


While price still plays and important 
role in buying decisions, more customers 
demand high quality at any price level. 



Quality more important 
than price 

1988 

80% 

1978 

30% 


- Leading quality consulting firm in Mass. 


Time 11/13/89 
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COORDINATING COUNCIL MEETING 



NASA Center for AeroSpace Information 



COORDINATING COUNCIL MEETING 
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STI PROGRAM COORDINATING COUNCIL 



PRESENTED BY NASA LARC 
GEORGE RONCAGLIA 



o 

CO 

CO 

LU 

O 

O 

< 

LU 




• NASA OUTPUT CAPABILITY OF HANDLING R&D AS 
WELL AS R&PM CHARGE-BACKS FOR SERVICES 


STI PROGRAM COORDINATING COUNCIL 


CO 
I — 

z: 

LU 


o 

o 

Q 

LU 

X 

O 

Li- 


CO 

CC 
LU 
Q 
CC 

O 

LU LU 


LU 


>- 

CC 

LU 

> 

I- 


o 

LU 

O 

LL 


0 

DC 

O 


O 

z 

O 

0 

z 

CC 

LU 

Q 

DC 

O 

>- 

CL 0 

Oz 

05 

LU QC 

a 


l- 

O 


LL 

O LU 
U_ LU LU 

ODC m 

LU << 
Zu 1 


LU 


oq 


£= CO 

0LL 

z< 

WLU 

<o 

LU Z 

!>-< 

§UJ 

O LU 
ZDC 


ce 

ce 

z 

o 

h" 

Z> 

eg 

cn 

coQ 

S > 

iiu 

*> CC 



CD 

CC 

I- 

ce 

Q 

z 

o 

LU 


g 

LL 


AM COORDINATING COUNCIL 


CO > 

I — o 

Z LU 
LU > 

O a 

O DC 
Q CO 

co d 

UJ CL, 

£- ofe 

O ° LU 
O Q Z> 

DC DC O 
LU < LU 
n IE DC 


LU 

O 

LijO 

a> 

H 

doc 

Z) LU 
1 CL 

Oco 

i — 
UJ 0- 
=>2 
Q LU 

rr\ H 


fc< 

Si 

LU H 
CEO 

S§ 

<£ 

-J 0l 

LU LU 

O DC 


CO 

LU 

CO 
LU - 

XX 

Ob 


o 

cr 

co u - 

CO 

< LU 
Q CL 

2 O 


< LL 

CL O 

d £ 


Z CD 
LU < 


h- ^ LU 
Z>Z> 
Od< 


L_' " 


RECIEVED BY CENTER AFTER A PREVIOUS 
INVOICE CAUSE MAJOR PROBLEMS AND ARE 
NOT AN ACCEPTABLE METHOD OF BILLING 
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RECON Input vs. SCITECH Output 

1980 - 1990 
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MECHANISMS TO HANDLE BOTH R&D AND 
R&PM CHARGE-BACK FUNDS (IF NOT ALREADY 
IN PLACE) 


Database Quality: User Views Test Producer Perceptions 

Barbara Lawrence 
AIM 

July 1, 1991 


A presentation to the NASA STI Coordinating Council 


Quality has always been a hallmark of the NASA STI database 
However user expectations for databases have changed, as has the 
technology of database building and information retrieval. This 
presentation reviews our understanding of current user expectations 
for a quality database and tests the NASA STI database aaainst these 
expectations. User needs are briefly characterize^ as areuser 

cti'JS* u abase i ualit y characteristics. Specific aspects of the NASA 
STI database are then discussed. The conclusions are that there is 
room to improve database quality and that, as a multiparty-built file 
cooperation and communication are key to database quality. 





Presented by: Barbara Lawrence AIAA 




Handouts for: Database Quality 



Users: Quality Needed 




■ Intermediaries 

• Detailed needs 

- Push producer and host 

■ End Users 

* Want Info, not process 

- Telecommunications the hard part 

■ Need guidance query formulation, 
database selection 


OotatMM Quality - AlAA 



^UsersTouaMt^^eededl 



■ Aerospace Scientists and Engineers 

- Understand concepts of databases and 
retnevaJ 

* Not intimidated by technology 
Need problem solving information 

- Need it on their time table 



OtfSbaM Quality - AlAA 





■ A itch i son: 1988 Mites Conrad Lecture 

■ Potzcher: user needs 

■ Max on- Dadd: DIALOG experience 

■ Mlntz: a thoughtful user 

■ SCOUG: a user group 

■ NFAIS : task force 


C Quality: Who's talking? 
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| Handouts for: Database Quality 


Users: Information Retrieval 
Environment Changes 


"N 


■ Expectations change: a history 
- having a database 

• errors 

• indexing 

• complete review 

■ Effectiveness « use of Info, as well as 

usefulness 

■ Retrieval system Is part of the 
equation 


QueUly - AIAA 


Quality Characteristics: Generic 



■ Accuracy 

■ Comprehensiveness 

■ Consistency 

■ Timeliness 

■ Accessibility 

■ Support 

■ Cost/Value 

■ TQM 



DtUbmm Quatty - AIAA 



Data Quality and the NASA Database 


■ Accuracy 

• Errors corrected on RECON 

- Errata, not handled 

• Data elements complete 

- Errors introduced in transformations 

■ Comprehensiveness 

- Covers all literature types 

- Source evaluation and selection 

• Has been resource limited 



Abstracts truly Informative 
OaS«« Qualify • AIAA 


y. 
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Handouts for: Database Quality 


Data Quality and the NASA Database 

■ Consistency 

- Thesaurus 

- Indexing 

- Codes 

• Subject class! Hcatlon 

• Hot topics 

■ Timeliness 

• Suits the literature 

- Currency varies 

Dotoboea Quality - AIAA 


[NASA STI DATABASE: Data Elements 
with Quality Issues 


■ 

Authors 


■ 

Author affiliations 


■ 

Subject categories 


■ 

Country codes 


■ 

Note/citations 


■ 

Typos 


k 

Doftobao# Quality • AIAA 
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NASA STI DATABASE: 
Fields to Consider 



■ 

ISBN 



■ 

CODEN 



■ 

Treatment Codes 



■ 

Specialized indexing 



■ 

Pre 1972 abstracts 



a 



L 
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J 
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Quotation 


& Tha quotation mark* In this quotation ara 
48 point Zapf Dingbats and tha taxt Its# If is 
24 point Halvatica Bold. Wa usad tha Labal 
tool to craata saparata Zapf Dingbats so wa 
could position fham without extra spaca 
batwaan lines. 


» QuaMy • AiAA 
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